New methods for creating testfiles: Tuning enterprise search with C-TEST
نویسندگان
چکیده
An evolving group of IR researchers based in Canberra, Australia has over the years tackled many IR evaluation issues. We have built and distributed collections for the TREC Web and Enterprise Tracks: VLC, VLC2, WT2g, WT10g, W3C, .GOV, .GOV2, and CERC. We have tackled evaluation problems in a range of scenarios: web search (topic research, topic distillation, homepage finding, named page finding), enterprise search (tuning for commercial purposes, key information resource finding and expertise finding), search for quality health information, automated bibliography generation, distributed information retrieval, personal metasearch and spam nullification. We have found in-situ, in-context evaluations with real users using a side-by-side comparison tool [3] to be invaluable in A v. B (or even A v. B v. C) comparisons. When a uniform sample of a user population uses an n-panel search comparator instead of their regular search tool, we can be sure that the user needs considered in the evaluation are both real and representative and that judgments are made taking account the real utility of the answer sets. In this paradigm, users evaluate result sets rather than individual results in isolation. But side-by-side comparisons have their drawbacks: they are inefficient when many systems must be compared and they are impractical for system tuning. Accordingly, we have developed the C-TEST toolkit for search evaluation, based on XML testfile and result file formats designed for tuning and lab experiments. These testfiles can formally specify:
منابع مشابه
C-TEST: Supporting Novelty and Diversity in Testfiles for Search Tuning
Tuning a search facility such as a Web search engine, or an enterprise search tool deployed in a particular organisation, is an economically important activity. Intuitively, an important end goal of tuning should be to maximise satisfaction across the searchers who will use the facility. Tuning should therefore use an unbiased sample of actual search requests, a judging process accurately model...
متن کاملA Monte Carlo-Based Search Strategy for Dimensionality Reduction in Performance Tuning Parameters
Redundant and irrelevant features in high dimensional data increase the complexity in underlying mathematical models. It is necessary to conduct pre-processing steps that search for the most relevant features in order to reduce the dimensionality of the data. This study made use of a meta-heuristic search approach which uses lightweight random simulations to balance between the exploitation of ...
متن کاملLabVIEW implementation of an enhanced nonlinear PID controller based on harmony search for one-stage servomechanism system
This paper presents a practical implementation for a new formula of nonlinear PID (NPID) control. The purpose of the controller is to accurately trace a preselected position reference of one stage servomechanism system. The possibility of developing a transfer function model for experimental setup is elusive because of the lack of system data. So, the identified model has been developed via gat...
متن کاملایجاد نیمه خودکار مشاپ های سازمانی با استفاده از توصیفات معنایی
Mashups are next generation of web applications. A mashup is a lightweight web application that is created by combining information or capabilities from more than one existing resources to deliver a new and integrated experience to the user. Mashups introduce a new class of integration techniques in enterprises for implementing situational applications (i.e. applications that come together to s...
متن کاملA Differential Evolution and Spatial Distribution based Local Search for Training Fuzzy Wavelet Neural Network
Abstract Many parameter-tuning algorithms have been proposed for training Fuzzy Wavelet Neural Networks (FWNNs). Absence of appropriate structure, convergence to local optima and low speed in learning algorithms are deficiencies of FWNNs in previous studies. In this paper, a Memetic Algorithm (MA) is introduced to train FWNN for addressing aforementioned learning lacks. Differential Evolution...
متن کامل